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DIGITAL MEMORY 



Hie piesent mvmdcm relates to a digifdl memory. 
Ths increased use ofconqmter systems aiKl digital dcct^^ 

itumbsr of diflteexxt memory stnictuies bemg proposed. .The wide range of 
applications xequiiiog a digital xoemoty mean Hiat memoriBs Iiavitig dif&zent 
properties are required. 

For example, memories &r iiss in a conventional stored program computer require 
higji levels of data integrity^ a single incorrect Wt wfll cause the computer to 
malfunction. A conventional coniput^ memory reqizirss that a read address is known 
ex£LctIy for re(!ri&val of infomiation stored at a particular address. Ttns is because a 
conveiitional memory stores a string of N binary digits Oiereinafter a word) a± a 
specific locaiion wilMn die memory. Only one word is stored at a particmlar location 
at aparticular time. 

&L some supplications involving tbe stora^ of large quantrdes of data some degree of 
enor in data retrieyed from the quegEXiory is acceptable* and can be ovedooked by the 
system. Fnrdiennore^ in some cases it may be necessary to aDow retdenral of data 
using an inexact read address. Ibe present invention is codocemed with a myemary 
suitable for sucb ^Hcations. 

One known memoiy stmctiire has been developed by Kanerva and is described in 
Kanerva P., ^'Sparse Distdbnted Memory**. MU Pre^, 1988 and also in KflUftfa P., 
*'Self-pK)pagating search; A Unifisd theory of Memor/', Report No- CSLI-84-7, 
Stanford University 1984. Here a large niimbet> of address decoders are cotmi^ted to a 
data memory coxcprising a number of word lines. Each addrejss degoder is cozmected 
to a different word thie wiHiin the data memory. Bach decoder is idratified by an 
address in Hie fbrm of a large binary number. 



{ 

2 

Whfin^thig data to the meoiory, an input addre$$ is pxesexxted to fiie set of decoders, 
and the decoders liaving an Identiilef within d. piedeteqznned Bjansxxms distance of 
the ii^ut address are activated. The wyrd lines connected to the activated decodess are 
flien selected to ineceive vnite data. When data is to be recovexed &om the meopuoiyt an 
ad dres s is presezited to the decoders, and a nmrbor of lines are again activated. 
Summing each bit of each activated word line^ and applying an a p pro pii ato tiueshold 
aHowsdatavvtitten tothememoty tobezecH^v^B^ and letdeving data in this 

way allowfii <iie effects of address and data errozs to b e nnnimised. 

The memoxy sfxucture pcoposed by Ksa&cv& has a number a disadvantages. One of 
these disadvantages is that Hannntrig distance calculationa reqqiice specifically 
designed haidware if tibey are to be canied out effidenfly. 

An altein^ve memoiy stracture which seeks to oveccome this disadvantage is 
described in WO 90/04830 (Uiivez^es Space Researdx A^ciatiooa). This system 
takes the basic arobitecttire proposed by Kaneirva:i but det^mines which address 
decoders should be activated in response to a particular input addi^s using a diff^^t 
cdtenon. In this systsni, a. subset of (1 bits of each decoder address are assigned 
specific *0* and ^1' values. When an input address is presented to the systent, the 
values of the q bits of the presented address are compared with those of each address 
decoder. £f the presented address has tlie same values at these q bits as a particular 
address decoder thst decoder is activated. 

It is an object of the present hivention to provide an altsai;$tive memory strusoture to 
those described above. 

According to a first aspect of the present invention, thfiie is provided a memoiy 
configuration for use in a computer syst&tuy the memory conspdsing a pluraliiy of 
address decodexs each of which is allocated an identifier having a, predetennined 
number of bits, each bit having first and second selectable states^ and a data msnoiy 
having a plmality of word lines of predetexmined length, each of the $aid addt^ 
dccodecs bdng activalable to select ona of the phnslity of word llne$^ and tho address 
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decoders comprising means to receiva an input addiess Iiavitig a predetermined 
Bumbor of bits andjinfiaxis to co mpai e the identifiear of an address decodei: wjXti the 
mpat addxess wba:^ tihe momoxy further coioprises means to activate an address 
decoder if at least a predetennzoed nrizttmiitn number of bits set to the first sdeotaUe 
slate iff. the xnpat addr^s carrespond to bits set to £he first selectable state in fhe 
decoder identLfiec j 



The benefits and adyantages proYided by the present invenfion, arise in part finm £he 
activation of addres^ decodes^ on the basis of bite set to the first selectable state in an 
input address^ whij9|bit8 set to tike second selectable state are ignored. In contraBt, the 
prior art systeam d^pciibed aboire activate address decodeta by comparing bits set to 
fEie first and sefxmd ^electable states, 

i 
I 

?Z€&srab!()f, the meaijs to oompare the identifier of an address decoder with the tnpirt 
addiess ct:H:isidBKs positional correspondence betvfrera bits set to lSy& first selectable 
sate in the input address and bits set to the first 'selectable state rn the decoder 

ideanitifieiES. * » 

Bach address decodpr identifier m^y have an aqual ncunber of bits set to the first 
selectable state. The means to receive an input address may be configured to receive 
addresses containing a predetermined number of Mts set to the first seleotable st^e. 
Suitably, the predeteirmined mnnber of bats set to ^ fir&t selectable state in an input 
address is equal to ipie number of bits set to the first selectable state in each of ihe 
address decoder idenjtifiers. 

I 

Preferablya the data jaaBmory comprises a phrrality of single bit memaries» sucji that 
each bit of each woi^d line is stored in a single bit m^ory. The data memory may 
comprise a data inpu^t line containing an equal monber of bits to each of the plurality 
of word lines. The mjemory configuration ip^ further comprise data writhig means to 
copy data &om the data input line to word hues activated by the address d^x^ders. . 
The data input line may be configured to receive input data contaiDiiis a 

pxedetetmined zimnb^ of bits set to the first selecteble state. 

I 
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Tb© memory configuration may fortlier comprise means to sum valuos stared at each 
bit of word lines seated by an address decoder to generate to adivaiion level value 
fer each bit. The memory coixfismtion may further comprise megjie to generate an 
output woM COTtaiaiing the predetranined number of bits set to the first selectable 
state* The bits set to jfirst selectable state in the ou^ut may be i piedetennined mimber 
of bits having the highest activation level. 

Hie memory may implenaented using a plurality of artLfidal neurons connected 
togerfher to firon ^ neural netwo*, The plurality of address decodeacs may be 
represented by a pliralily of address decodear nenrons and fixe data memory may be 
loeprjfesezxted by a phijality of data neurons. 

t 

The inveotloa fiaDtOi^r provides a neuxgl netwodc losmory configuratiain for use in a 
computer systeoi. the mwaary conoprisins a plurality of address deooder neuroc? eadi 
of which is conn^cCed fo a predebennuied niunber of input nranmSi and a data 
memory having a pljiraU^ of data neurons, eadi of the said addFcss decoder naurons 
beiAg activatable to! select some of the plurali^ of data neurons^ and tt^ address 
dec Oder nevnms coxnptiGirig mrans to reooive a signal representing a firing of an isfiut 
neuron to wMch it ifi|comiected,\vh6ri^ decoder neuron camptisesmeazis 

to activate d^ neajjons if firing dgpals are received &om at least a pr^eteacmined 
minjEmom munber oj^inpnt oeuiras to which the address decoder naqaron. is cssnnsctedf 
! 

In practice, evay ai^dress decoder neuidn may be connected to every input neuron, 
widi each of these jconnectiona beii^ allocated a or "0* weij^t In sudx an 
inqflementBEion a '0'| weigi^t has the same e£Ebct as no connection between a particular 
input neuron and ad^ne^s decoder niBuron- Thus only firing sigtJ4l$ from input neurons 
connected to an addpess decoder nemacm such tiiat iJbie ccamection has a *1' weigjit, 
wtU rspresent a cotin^»ction aa described in the previous paragraph. 

Accoiding to a secoi^d aspect of the present inveatioii, there is provided a meihod for 
operating a mflmory ^r use ia a computer systeiji, the memoiy comprising a plmalcty 



I 
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of addiess decoders eadx of which is allocated an ideatifier having a predetemmied 
number of bits, eacb bit having jBrst and second selectable states, and a data memory 
having a jtolity of woid lined of predetermined l&OLgjStk each of the said address 
decoders being activatable to select cac of the plurality of word lines, whisem an 
ioput addr^s having a predetennined zinmber of bite is inpirC to the address decoder, 
the Jdentifier of an address decoder is compared with the input address and address 
decoders are activated if at least a predetfflnined mininium nmnber of bits set to the 
first selectable state in fhe input address correspond to bits set to the fbrst selectable 
state in the decoder identifier. 



The input date, may be presented at adataiiqrat of the data memory and the data may 
be written to word lines activated by fhe activated "address decoders. The 
predetermined mininxum zrumber of bits is preferably set such that fewer fhm lOO 
address decoders are activated by any valid inpurt address. The ptedetennined 
minimum number of bits is more preferably set such t3iat less than 50 address 
decoders are activabed by any valid input address. Most preferably^ fhe predetermined 
minimum nxnnber of bits is set $uch that no more ibam 20 and no fewer than II 
address decoders ate activated by any valid input address. 

According to a third aspect of the pzeaent invraitian, Ibere i$ provided a mediod fbr 
optimising the opecatian of a conxputisr memory wiaicli comprises a {ilurafity of 
address decode each of v/bi<iL is allocated an identiSer having a pred^scmined 
number of bits, each bit baving first and second aelectabld states, and a data memoiy 
having a plurality of word tines of predetsEinified Lengfh, eadi of the said address 
decodefrs being activatable to select one of the plurality of word lines^ the niBmory 
further comprisjng means to receive an ix^nft address and means to activate ono or 
more of the addns^ decoders if a compari$on between a decoder identifia: and the 
iqpttt address exceeds a predetennined comparison threshold; 

the mefiiod comprifiicg deterooining an operationally beneficial nundier of 
address decoders to be activated in re^onse to a valid input address, and configart^g 
the comparison threshold such that a valid input address will activa^ a nuoiber of 



addles decoders substantially equal to the operatLDtially beneficial number of address 
decode to be activated* 

Hie conipaiisoiL may compare fhfi number of bits set to the first selectable state in the 
ixxpat addiess ivifli number of bits set to ffae first selectable state in eacb. of the 
address decoder ideatifieis. 

The aperadonally beneficial mmibear may be deteaninfid so as to allow m axii wn n 
epror fiee date recovery &om ifae data memory. This maybedoneusiz^ a fimction of 

^where: 

w is the operationally bensfioial nambBr of address decodecs to be activated; 
h is the probability of an arbifraiy bit in the data memory being set to the first 
selectable state; 

i> is the number of bits in each word line, and eadi word line has li bits set to 
the fast sdectable static and 

Pc is the desired probability of a bit being correctly recoveted fiom the data 

stxHie; 



The function may be: 

The operationally beneficial number of address decoders to be activated may be 
detemiined so as to allow maximum error free data recovery, while allowing some 
fToti^ to be recovered with ecrors* The operationally beneficial number may be 
determined using a function of the formt 

where: 

14^ is the c^erationally beneficial xunnber of address decodezs to be activated; 
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A b the piob*iHt5r of m aibiiiary Isit ia tite data memory Tsdng srt 

selectsibiB state; 

iJ k the mmber of bite ia each wrd line, and each 

flte first selectable state. 



Tba fimotion may be: 



w 

Preferably, the operationally benefidal number w is set so as to have a value greater 
f h^ ftiat givfen by Htc equation: 

w 

and less than that givea by the equation: 

The inventian fhrther provides a earner medium cancyiog conqjutw readable code 
means to cause a computet to execute proceduie in aocoidance with liie meibod set 
out above» mi a compoterprogram for oairying out Hie melhod set out above. 

An embodiment of liie present invention will now be described, by way of example, 
wifhrel^cmce to &b accompanying drawingB, in which: 

Fieore 1 is H Br?h«m«fa'e iltnatratinii of a menwrv oonfignration known flom me prior 
art; 



KgoTB 2 ia a schematiD fflnstradcai of amBmoiy configoiatiaa in accoidance with the 
present itnrendon; 
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Figme 3 is a fhree-^Hmensional plot showing storage capacity for a partLoular 
oonfigaration of the memoiy iUustrated in figure 2; 

Figure 4 is a ocsatourplot of the threeniimsasionalplot of figme 3; 

Egare 5 is iJie ccmtour plot of Figura 4 amended to take into accoimt indqpendentbr 
aetivBtBd addr e ss de co d e ra; 

Figme 6 is a adwiijaitiB ilhisfraliott of an 

to tmpTgmfipt amemory in accoidance with ibo present xnvenlion; 

Figure 7 is a schemaiic ithi^ration of a first configuratioii fbr cannsctions hetween 
ii^ namms and address decoder netnons in an artificial neural netwoik having a 
goural &rm sirnilar to Ibal; of figure 6; and 

Figure 8 is a sdiematic ilhwlistiott of an alteaoiativB configuratkm Ibr connficticms 
b^ween input neatona and address decoder neunms to that shown in Flgote 7. 

The address configuiation described by Kanerva and outlined above is iOnstrated in 
Figure 1. A plurality of address lines 1 are provided on which an address can lie 
traasmitced. The address is a binary number and each address line 1 transmits one Int 
of the address sudh lhat n lines are required to represent an n bit address. Ibo addrras 
liaes form input to an array of address decoders 2, and eadi address decoder 2 has an 
identifier in the fenn of an n bit bmary number. The address decoders 2 are 
connected to a data memory 3, which con^jrises a number of word lines for atoriiig 
data. au5h word line has a storajge capacily of M bits. Each bit of each word line is 
stored m a counter 4. Data to be written to 1h© data memory is transmitted oai a data 
input line 5. Data read ftom the mfimory passes along data lines S to adder blodks 6 
and is oulput :&om Hie adder blocks 6 on data, ou^ut Ibies 7. 
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Typically, the number of addiess lines 1 is of the order of one thousand, such fliM a 
one 1h.oasand bit binary nmnbor is presented to fiid address decoders 2. Since amy 
large nimiber of address^ can represented by a one thoumid bit binary number 
(approsiiOTtely itisnotpracdoaltohaYeaseparatewordlinB to store data ^ 

each possible address. Accordingly^ a large random sample of addirases wilhin the 
possible address ?pace are implemented- Typieally one TniHion addresses are 
implemented* and each of these is allocated to an address deooder 2. It is for this 
leason ihat the memory proposed by Kanerva is described as q»arsB» since the one 
million (10^) decoder addresses are ^acsely disfiibiited within fbe available addrcra 
space (10^^' addrKses). 

Innse, an address is transmitted on the address lines 1 and is lecdived by the ad^ 
decoders 2. A comparison between the transraittwl address, and the address of eadi 
ad^ess decoder is made- This comparison is perfoimed using a Hamming distance 
calculation, which computes how many bits of the transmitted address arc difierenl to 
the addresses associated with each address decoder 2. The Hamming distance 
calculation is presented in equation (1): 



where: 

X is abinaiy vector having N dements; 
y is abinary vecfor having N dements; and 

H is the TTqniTTnfng distance betweeax and;;, bong the number of bit positions 
atwlucSx Xi^yi. 

Decode!^ witbin a predetemiinfid Hamming distance r of the transmitted address x are 
activated* IWs region of activation tat each address may be viewed as a sphere 
cerdxed at x and having a radius r. For example, Elanearva has shown that if each 
address has 1000 bits, and fho predeteEzcined Hammmg distance r is 451 then the 



Spherical regioii of acthration contains about rrrjrOf fbs addresses associated with 

address decoders. In use» a processor compufces a Hanuning distance between each 
decoder address and a preseaited input address. If this distance is less than r fer a 
partioular n^^y^py dstcodBc^ then that the decoder is activated. 

Details of a process for writing data to memory, in the system proposed by Kanerva 
will now be described. When a given address decoder 2 is activated, data presented at 
fhfi data xnpGt 5 of the memory is written to the word liiie which is connected to the 
acthraied address decoder. Smco activation of an address decoder is initiated 
i^irihenever an address occurs which is within tiie predetennined Ha mm i n g distance of 
thai address decoder, flic data will he written to a numljer of di:ffereDt word lines. 
Similarly, there ma a number of diffearent addresses which will which activate the 
same addcess decoder. Tlaia means that data \vill be written to a partico^ 
fi)r a number of difiEereot input write addresses. 

Eaich word line con5)rises M counters, each of which stores a single bit of data. Tto 
cocinieis at a given word line allow linear accunmlation of a xamiber of M-bit words 
Stored at that word Ime, If a '1' is presented at a particular bit of a word line, the 
OOiTCspondiDg counter 4 is injcrecneDted. If a '0' is presented at a particular bit; the 
counter 4 associated with, that fait is dficremented. 

It should be noted that the cottnters will have a finite cq>acrly, and once ftis capaoity 
has bean reached throu^aiK»e&8ive data mhns, fuxOier writing of data will have no 
ejBEbct.. For escample. if the counter lias teadlied its maximum vahie ihroq^ successive 
writing of *l's , writing another 'I' to that counter can have no eSbct. Similarly, if ihe 
counter has readied its rnfnTTmmi value timou^ successive wrifEOg of '0*s , writing 
anofbsr '0' to that counter can have no eEBscL It is usually sufBcient Usat S bit 
counteis are used^ eac3i having a range of dbl27. In practice, it may be sofSdieat to 
impl^ent 2 bit or 4 bit counteirs. It should be noted however that up-down counters 
of any bit length are a consideiable ov^^head in the memory proposed by Kanerva. 
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The address decoders 2 are configuied sudi that for one mOIlon address decoders 2, a 
random addre^ is within the predeteniiined Hamming distance of about 1000 of the 
decoders. Thus, for a random input address, mpvt, data will be -written to 1000 word 
lines wiftan the data memory 3, liie aiqpiopriate counters 4 of each word line bdng 
mcTCTrreated or decremented as described above. 

Details of a process &r retderving data &om memory, in the systenoi proposed by 
Kanerva will now be described. 

Data is retrieved fiom tbememory by providing an iisput addr^s onfiie addms lines 
1, If this inpat address is idsnlical to that used for writing the data that is to be 
retrieved, the same address decoders will be activated, and data bam the ccaxect word 
lines will be xelxieved. 

If an address close to the address used fin: writing is presented, many of the address 
decoders 2 activated Id write the data wiU be activated to read tihe data. However, it 
should be noted that some decodes aotlvaied for writmg will not be activated^ and 
othem \^ch wiere not previously activate will now be activated. This results in noise 
being present in the retrieved data. Assuming that there are a sufficient muuber of 
decodes ED common between the read and write procedures^ the correct data is still 
output &om the adders 6, and Ihe effects of noise are mitigated. This in an 
advantageous feature of the memory described by Kanerva. 

The values held in the coimters 4 of eac^ word line connected to an activated address 
decoder are output onto dat& Unes 8 during a read cycle. The values output on each 
data, line S are added by an adder 6. The adders whose output exceeds a predetermined 
threshold genearate an output ^1* on the output data lines 7 and those which do not 
^KCeed the threshold genecate an outjiut *0^ 

Kanerva has determined that (he structure and funcficn of the memory means that of 
Ihe one ihousand. bits in an input address only six hundred bits need be cottect in a 
read address to aSow successfiil retrieval fi:om that same input address. For example. 
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if a bit is set to *V in one liiousand of the wotd lines daring the writing of dala, it 
would be expected tihat ttie adder 6 connected to the comrters holding values fiir that 
bit woidd output 1000. However, if only a majority of tiie one thousand word lioes to 
which data wa^ written are retrieved fay a retrieval address, <he adder will in gaieral 
oixiput a value which will be intaipreted correctly, TIius. the manoiy jooposed by 
KftoBTva has usefiil error correction. proportiBs and allows the dB&cts of noise 
described above to be znitigated. 

An N-of-M code is a code ia which N bits of an M bit binary number are set to '1'. 
Any N-of-M code fluBrefore comprises N bits set to *1% and M-N bits set to *0'. Each 
vahie lepicseaited by the code difEbca in the orda: in wHdi ftie '1 's and *0's ^pear. N- 
of-M codes aie self timing in lhat when N '1 *s have becsn rec^ved it is known that the 
code is complete a vahic has been receivedX but if less lhan N 'I's have been 
received it is known that the code is inoamplete. 

The menuny configraatum of the presaA mventioa takes the sparse distributed 
propBTtae&of the prior art memory of figure 1 and introduces several i^jpBcations of 
N-ofJ^ coding so as to provide the memory wifli a mmiber of oihancenjents wUch 
ai« set oat in further detail below. A memory configaialion whidi rnibo r rie s the 
present invention is iUustraled at a high level in figure 2. R can be seen tiut the 
goMcal oonfigmtttHni is amilBr to that of fignre 1, and Ubae xe&iieace mnoerals are 
-BSed where ^propriaie. Only da&amces bdweea &e manoiy of figure 1 and Hat 
provided by the invenlian win be discussed in fmtiiec detail here. 

In general, the raflmaiyconQpElsra ^''address decoders 2, w of Trfnch are activated fbr 
apanicular read or write. Badi of the JF address decodes cosaDprises aa^ bit address 
and each of these addresses comprises a Wts havmg a value 'l'. Thus, the address of 
each address decoder is an »of-A code. When an address is input to the address 
decoders 2, it has i bits set to a '1' vahie and bits set to a "O* 'ualue. Thus anyiiqnit 
address is an j-of-A code. In soma implementations it is the case 4at all txqrat 
addresses have a number of bits equal to that of the addresses of the d^ders 
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(^). Howev^. there is usually a marginal benefit in allowmg i to be independent 
fixm a. 



When an input address is received by tiiis address decoders 2, a comparison between 
the a-of-A code of each decoder, and the i-ofA code of the input address is 
pex&zmed. Bach addisss decoder lias an associated threshold 71 The tbxeshold T 
detaaxiines tbe mnnb^ of I's wldch mnst ooxnoide betwera. the isspat address and the 
address of the decoder. 



Sin^lified examples of the decoder acdvation procedure wiU now be pr^ented In 
these examples^ it is assumed that thcsce are three* address decoders (A, B and C), each 
bavins ^ 3'^£-8 code, The addresses of these decoders axe as follows: 

A: 01001001 
B: 10001010 
C: 01101000 



In the descirption tbat follows, each of the addresses is indexed firom 1^ to xigh1;> 
starting at an index 0. 

]h a first scenario, the threshold for each decoder is such that at least two bits of an 
mpnt addled must coincide Tviih two bits of the decoder address. Tims when 
presented witibi: 

Addressls 01001000 (a2-of-8 code) 

It can be seen that decoders A and C are activated* as bits 1 and 4 of liie input address 
are '1 's and bits 1 and 4 of the addresses of decoders A and C are also 'I's. Thus the 
threshold of two coincident bits is achieved. Deco<ler B is not activated, as only one 
of ' 1 ^6 is coinddeat O^it 4} betsveen Address 1 and decoder B. 



Similarly^ wbesi presoated with an addcess: 
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Address2: 01000011 (a 3-of-S code) 

tod assuming that the Hireshold T temsins unchanged, it can be seeo ihat Address2 
hasn-smbhsIandTand decoder A has * rs in the same poatians, meaning that 
decoder A wUl be activated. Decoders B and C are not activated, as only one ' 1 ' bit of 
each decoder addrasB is coincideot with a '1* in AddiBSs2. 

the activation piocednra described above acftvates w decoders ftom a total of m 
TtaM. tho aetivatLoa pattern is a w^f-TTcode. This is similar to the process described 
ilwvcvrithieftaaioete figure 1, althou^ it iB important to note ^ 
preferably set so as to ensure that an optimnm or near optimian of number of address 
decoders fire during ft filvea read or write ^^yftle. This optimum number has typically 
W found tD be fer teps tibai^ Uw number activated in the memory proposed by 
Kanerva. Hie nxvoitor has realised that data can be read fiom or ^mttea to the 
memory with sufGdent accuracy usmg lesser number of dsooder ^activations, and tort 
activating oas thousand decoders, as in Kanerva^s system, is imnecessaiy. A more 
detailed analysis of this ohaiaoterisfio of ihe memory system is presented bdow. 

llw w activated decoders in turn sdect w word lines to 
^-written to. Tlins, the comimnacafi0n between die ad^ 
lines of fiha data memory 3 is a Mvof-IToode (i.e. w aotrvateddeoodos of r decoders 
in all). K should be appreciated Ihat it is not usnally practical to ensure tiiat w is 
constant, given tiie activation procedure involving ind^^eotly activated address 
decoders, as described above. The ccmmmmcatiim of tins of code ftom the 
address deoodets 2 to ttie date memoiy 3 is denoted by an arrow 9 in figure 2. 

The w activated decoders sdect v W lines. Write data is presented to write data to 
flis data memory 3. and read data is read fi»m tiie dala memory 3. The reading or 

writing U effected at each of flie K' activatad word fines. The input data 5 is 
considered to represent a d-of-D code, where each wad line oompiises D Wis, d of 
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wMch are set to *1 's. Given that each input is a d-of-D cod^ each write data qperBtioii 
will be writing data oontaimsg d U's. 

Vii^f^ti bit of each woid line is stored in a coov^ioiial binaiy single bit memory. All 
bits of all word lines axe initially set to * 0*. When a * 1 * is to be written to a bit, the "0 * 
value is replaced by a '1', Once a bit is set to '1* it is never reset to '0% even if a 
subsequent write cycle attMipts to write 'O' to that bit. That is once a data write 
operation has set a value, any ferflwr incaMse in this value is simply noise, and can 
theiefbre be discarded. This storage procedure is based on the nxiipolar nature of N-of- 
M codes, and the reasons for its reliability are givmi below. 

When data is to be i^ead fiom the data memory 3» an address is presented to the 
address decoders 2 which i^ts in w addie^ decode being activated. These w 
adders deoodfio^ in turn activate w word linra in the data memory 3. Each word 
Cttitains Dhha, andihe contents of each bit of each activated word Use are summed, 
to yield D outputs- These sums are herernafta' referred to as adivatum levels. It is 
known that all writB data is a dnaf-D code, it should tfaetefbte be die case that the data 
read from the memory is also a d«of-D code. - 

Jf a single write cyole writes a value to a particular addiws^ and fhfc same address is 
pres^ted for a read data operation (without an intervening furfhar write cycle) it caa 
be OJQieotBd that the activation level of each bit set to ' T in the write data will be w, 
while the activation level of each bit set to *0* m the write data will be aero. Thus, it 
^be easily seen which bits should be set to '1* in the output data. 

However, if two write cycles write data to different input write addresses, it may be 
that some of the address decoders activated during the first write cycle are also 
activated during ihe second write cycle. That is, some word lines within the data 
memory 3 will oontaia some bits set to as a result of the first write cycle and some 
bits setto '1* as aiesrdt of the second write cycle. 
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•When an adibess equal to the first ■write adcteess is presented to flie address 4ec!odera 
2, the read data pwcedure activates w word lines, and calculates activation levels as 
described above. Given ttie action of the intervening write cy&le, fliese activation 
levels Will contaic some values derived fiom the second writ^ that is some bits which 
wexe set to *0' in the first write cycle will now hsve a non-«eio activation level, 
iMhoTi^ these acdvadon levels idioTddbe leas tothat of the draingthe 
fiist wri.tB cydb. That is, more ibm rfWls wll have non-^o activalian levels. 

As it is known ftat the output data should be a d-of-D code, a modified wijmer takes 
aU (WTA) alBoriflno is ^Ked to detennine die d bits having the highest activation 
leveL These <f bftsaie those wM<^ were most likely se4 to *1* during the writmg of 
dato to that address, and are suKxirdingly set to ' 1 ' in fhfi ottlpot data. 

A WTA algoriflim jwanally detranrines a single winner fto«n a collection of data, to 
Has case, the modified algorithm, (hficdnafc^r refiaied to as a h-WtA algoriflan) 
deteonines d values wMch have the acdvatioa levd, and the bits Jusviag fliese 
values are set to T in the oulpnt that is, these bHs are dis -Swimfiis" in die 
algoriflim. A WTA algoridan csn be in^lemcntcd by sorting the D vahies irang any 
amtaUfi sotting algorithm, and selectinglh^ 

It is important to note that the ^Hcation of N-of-M coding to ihe data nranniy 
aDows this fenn of read piooesa to be employed, as it is known bow many *1 's should 
appear in the output data. In the system proposed by Kanetva. where data written to 
the memDiy is binaiy. Uieie is no way of knowing how many 'Vs should appear in the 
oiilput data, and accordingly Kaoarva uses conntatp as storage dements to aooumnlata 
data written to each bit, fte contents of fihe corateis being summed, and a threshold 
b^ng applied to generate an output 

It was noted above, that a single bit memory is never inrananented past This is 
because the value at a particular bit can only ewar be '1' off '0'. Therefore, 
incremeniing past T must mean that some noise is stared at that bit which is 
ttodesiraible. This noise is generated by two write addrasaes activattog a oammon 
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word Una. la ^ pr^eot case^ Ibd A?-WTA alsoiiAm conecdy identic ^vUdb bits 
should be set to 4% WEttuHit the n^ed fixr ooimters whidi can. be incremeiitBd past 
Konerva'fS nziplmfintaiion of i]i>-d&mi comjters, and a zero finreshold is no laager 
necessary to deteimzne whether a particular ontpqt bit should be set to '0' or ^1 \ Thus 
tho pressDt invetdioa provides a m&moiy in which data is stored in conventiona] 
single bit monories tsxd not in counters. This results ii^ a great saving in 
miplementation cost. Furthermore, recent developments have resulted in large 
quanijlies of suitable memory (E!AM) being available ch^^ly and efflcieootly. 



A conrmnn measure of the effectiveness of a memory is its storage c25>aoity, in terms 
of the number of bits of iisfoxmation thai may be usefully stored for each bit of the 
memory- Using this measure a system embodying the present invention yields 
cotisidja-able benefits over that proposed by Kaaxexva, The main reason ftir fihis greater 
atoiage capacity mses from the writing of data to a smalls nfUmber of word lines. 
Typically, the inventor has discovered that while Kanerva wrote data, to some 1000 
word linesL» data can be written tD a &r smaller number of word lines while retaining 
the bfinefit5 of Kanearva's memoty, and obtaining greater storage efKciracy. For 
exiani^le^ for menuxty charaoteristics as de&ciibed below, it is prefetzed that data is 
written to less lhan 100 wwd lines; more prefixed that data is written to less than 50 
word linesy but operationaQy mostbenegcidi that data is written to between 11 and 20 
word lines. An analysis of storage o^aoity» tog^er wilh equations to deteimine the 
optimum number of wcxcd lines to which data should be written yf^ will now be 
presented. 



The data memory 3 Is a fiami of comelafion mstnx memory. Correlation matrix 
mi^ncines have been studied extensivdy izii fiur exa£)Q}Ie, Kohonen^ T, "NSotrdatiaxL 
Matrix Memories^, IBEE Transactions Computers C-21(4) April 1972, pages 353-359 
and Turner. M. and Auslin, J. ^'Matching Performance of Binary Ocmrelailon Matrix 
Memories", Neural Netm>ika 10(d), 1997 pages 1637-1648, An outline of this 
analysis is presaotedhrae as a basis fin: the derivation w. 



c 

It is dssnmed fh&t addiess and data iisgpute are unifonnly distributed over tbeir 
respective spaces* Tbs dat^ meoDOOiy 3 initially ccmtaiiis ''O's at each bit The measure 
of occopancy h is nsed^ where h zepieseuts tiie pmhability of m ^itrary bit in the 
data memDry benng set to ^1* at some point in the mite proc^ After Z mite 
operatioxis die expected occu^raticy is grraa by equation (2): 




The f *. i-1 teim of eqaatHm <2) gives flie probaibiBly of setting a particular bit 

Ijk d) 

wittrihtt fiw data, memoxy to * 1 ' in a ^ngje write operation. Snbtracttag tbis term from 
1. gives a value lor tbs piol>abiliiy of not setting liiat W to *1' in a partioolar 
operation. This siibtractioii is Hisb. raised to flie power of Z, so as to rqaeseot liie 

efifeot of Z write operations, nnjs the torn pro-rides a probatoflity of 

not setting iheariritraiy bit to *r after Z write c9er8fions.Tlnis, in order to derive A, 
&s piobabiKly of setting an arijilrary bit to 1 after Z operations, fhis tenn is siflrtrasted 
flom 1, to yield eqnafion (2). 

It can be seen fiom equation (2>, fbat A is initiafly 0, rinw 

aide of the eqiistion simplifies to 1 — 1. & ineieasea sionotontcally towards 1 as naore 
data is stared in ^ememoiy 

Eqiiation (2) can be zewzitten using logarithms to ^ve: 
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The term fjL.iL^is small, ghrea that df is typicaUy much azul>f^is 
\,W Dj 

^ioally much fimaDer than W, Thesefiaie thenatuiBl logarithm on the right h^f^ side 
of ft(0 equation caa be ^pprnKiznated to ths first f£zm of flie Taylor B&iss. This givi^ 
a3x ecquressiofa £>r Ite number of staxed data itsms, as ^ out m equadon (4). 



Z = -hia-A)-— (4) 
w a 



The total infomialion stored m fhe memoiy afier 2: write Cpeiations maybe ideidified 
by considering Oie number of symbols that can bi& i^prcseoted by a d-of-D code. Thi& 
ixifonnation conteot is givea in equation (5)> 

/(Z)=Z4os[(:^]bits (5) 

We can compirfe storaga efficieaoy by ccanparing the inloxmadon capacity givm in 
equation (5) with the number of bmaiy storage locations m die data memory to give 
the expression of equation 6. 



where tfCZ) is the storage efBoieaicy of the memory. 

For a large value of D. the combinatorial term of equation can be approximated 



For a sparse memory where d is much less than A substituting equations (7) and (4) 
into equation (6) yieldss afiqr simpIificatioiL: 



(8) 



^eie ri{Z) is the slDtage e£Scl6acy of tbe memoiy. 

This shows, fhat thomo^t ^Gldent mmioxy. for a gtvon occupancy A, and undec 26ro 
ettor conditfons is one where dala is written *l-hot' 
code);, aad wbeie fhe sniaU^ xnimber of address decoders fire £»- a given input 
addtess. That i5> rf=l and >v=l. However, setting d and vi/ ia aocoardarjca with tiiese 
ttitcrifl, will adversely afifect the lobustness of the memory imda: Wgh o^iacity or 
input emizs, afi is described below* 

In order to consid^ die robustness of the data saemoiy, the retrieval of a aongle data 
word^ following the wiitmg of Z data words is considered, where the retrieval address 
is presented without error. The w address decoders activated by the read addre^ will 
be the same w address decoders activated dwring the write data process. If the data 
written to a particular bit was a ' 1 % then each of the w word lines should store a *1' at 
fliSit bitp to give an activation level according to equation (9). 



where E(X) is the activation levsL 

I£ howev^, the data value written to aparticular bit was a *0% the expected activation 
level will be: 



iff) 



(10) 



vi/b&TQ A is as deSoed above. 
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Since h }$ less fhaa 1, themeaoiory canbe CKpected to recover the mittm data, sjnoe 
the bits set to *r vnH have a hi^isr activation level than those $et to '0* as oaa be 
seen from equations (9) and (10). E&wever, the activation level for a bit set to '0' is 
subjecc to statistical vstiaticm, and 'wifh scxme (small} probahiliiy, oil &e wei^ts "will 
have been set to T by othesr data ^tes. In this case, the activation level of an output 
diat should be 4 '0' voUtielbdSsaaieasthatofanou^uiwMdis^ This 
gives a possibility of a '&lse 1^ eizor condition. In the absence of wzite fflccora, it can 
be seen that the» is no posaibiHty of ^ftlse 0° enors. 

A 'fidse 1 * error happens only if eveay bit contribulmg to a '0^ output is incoirwfly set 
to *1 Thus, the probability of a '0' bit being canectly read is givea by equation (1 1). 

That gives a value fcr ths probability of the w bits contributing to the activation 
level having been iTXConecfiy set to *r. Thus subtraclixig Ibis value ifrora 1 gives the 
probabiKty of a vakiebebxg oonBctfy read, that is the probability iiiat the iv bits 
have iwt all been inccttrecfly set to '1". 

A data word is read correctly ifeacai of its D-d bits set to 0 are all read cocreetly- 
The ixEDbability of ooiiecEty teadxng a word is given hx equation (12): 

^(data woidcoirect) =11 - A"]^ (12) 

That is, the eaqpression of equation (11) xdised to the power of D-d, given that i?Hf bits 
are to be recovened in totaL 

If Z data words have been written to memory, tbe probability that every stored data 
word can be read corrBcfly is given by equation (13): 



(13) 




As h incieases^ the probability of error free xecoveiy decreases towards zero. 
Revmtmg equation (13) vtsm^ Ipgari&ms gives: 

Kii)=^-P-*0-lQa-A'') (14) 

SSince is very maall ttfherc w is considerably gi:eater ifaan 1, fixe first tsm of tt© 
Taylor series givQS a close &ppt(admB&an for ^ logsirithm on the zi^t hand dde of 
equation (14) to yield: 

= -Z • (JO if) - A'' (15) 

In order to optimally configure the data memoiy, it is oonsLdared IJiat ftie nDmber of 
address decoders I^and the word leagtbZ> are &ed. 

It is also consideo^ diat the data coding (d-of-D) is fixed. In order to maximise liie 
probablHty of correct data tecovexy, it can he seen fiom equation 13, that A^, or 
egoivalently (w hi A) should be minimised. Tfaits, we seds: a solution where: 



A(wlnA) = InA+:^-^ = 0 (1« 



Rfcalling equadon (4): 



TV a 



which can be j swritt en aa: 



an 
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Difi^entiadng equation (17) tidfii respect to w and substitnfxns a vahze of Z obtained 
firoxD. equadon (17) gives: 



Substitnling fbie valtie of — frcan equation (18) into equadcm (16) gives fbe 

dw 

fbllowine foxmula fbr h: 

faftH.^^- a-*)-ML-^>j ,o (19) 

Simplifynig to give: 



IiiA+i<-(l-A)liia-fi))=0 C20) 



multiplyiiigboth sides of eqciatimi (20) by by A.* 

*Inft.--Cl-A)-laa'-A) = 0 (21) 
Equation (21) lias &e sahiiioii A=QtS. 

It has praviously beea sho'wn th^t the maxinium infomiation content of a conelation 
xiadrix memory oocurs wheii fr=0.5 (See, Nadal, J-!P,^ and Toulous^ G., "lafomiation 
Storage in iSparsely Coded Memory Nets" Network 1 (1990). pages 61 to 74). The 
optlx&isatioii of ppesented above» was concexined not wiih how much informatioii the 
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memoiy can hold, but how much, can te recovered reliably. The maximum 
recoverable conisnt tbarafiue coincides with the mnvi'Tnitrti conteaiL 

Substituting a value of Z fiom equattoa (4) into equation (15) yields: 

f^-^.^^M^.W.^-iD-d) (22) 

Using the optintTmi vahie of ft darived above, liiat is h^3, and givea that ^ <t and Z3 
are known, and using an acoeptable ptobability of emsr fieo rBCOV«ay a value w 
canbeconiputed. 

However, in some caicumstancesi, totalfer error fiM 

too x9B<xkti.ve a lequicBmeDit, given fbs: nature of tiie menwiy. An altemativB 
definitioiL fbr information oqiacity of the nranoiy is to estabUslL the inaxi minn 
number of data items that can be recovercd wifhoirt error, allowing flsat some other 
litems may be xecoveced with ecrois. 

RecaUmg equation (12): 

^<datawoidconect)=[l-A''l^ (12) 

it is possible to derive an expression for the number of data items that can be 
lecoveted aeourately, wWle othais wiE be recovered with enroxs iSa This eiqpresaon is 
giv^ in eqvaSksa. (23)! 

S.-Z-El-An*^ (23) 
^t^ecB 2*19 Ibe iKiinber of woids Qontained in tiieme^^ 
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For a given value of ^ v is vaned so as to zriasimise iS^ and &i5 

IF data is "vmtten to tte memory vxsSi each new eoatry caases oilb or more additio&dl 
Tdadout eaoots, the Ty*flywnii»v* itseiiil capdcdty of the xnemaiy has "bem reachfid. The 
pokit is readisd whea: 



:^=o (24) 
dZ 



Rewrittuig equaticHi (23) as: 

hiE^^iD-d^-baZ-ll-kn (25) 
MkKXs&a&ae eq.uatioa (25) gives: 

E, dZ 2 (l-A*') 



iUswiitins equation (4): 



TvZ = -!na-A)-^ (27) 



it can be seen fiom equatiotx (27), that for ccinsteat K is constant, (gjlvm that A 
and ^ arefxed. Thus: 

^ = -Ji: (28) 
dZ Z 

SubstLtutmg equadon (28) into equation (26) gives: 
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(29) 



Equation (29) gives ftn alteraative vahis for optimum Wf, the object is to tetdeve 
as many wordd as possible conectly^ while refrievrng some words with wars. The 
value of M' whieh can be obtaixted from eqaation (29) is smaller dian that obtained 
from equation (^2). This teflecfs tfae &ct ttiat activating a larger nmnbear of address 
decoders (Le. increasing iv) r^ults in a memozy liiat has &mer anors bi3t loww 
opacity. ConvGcsBly^ a mallear value of iv r^uUs in a memoxy pocodudng more saass 
but having a laigar capacity as each write data opBration sets values in tcfw&r word 
lines of the data memoiy . In both case^ Uie maxiamm extractable infomxation occucs 
When^^^. 

The inventors having derived tbt6 scpresidcms set out aibove^ has detonnined that a 
memoiy embodying the present invesxdon can be much improved if v is set so as to 
lie be^een the values detesmined by equations <22) and (29), 

The analysis described above will now be desoibed with reference to figures 3 and 4. 
Figure 3, shows a three-dimensional plot of equation (23). An axis labelled Z 
represetits the immber items written to memory;, an axis labelled w axis represents the 
number of addrsss decoders acfivatai by a read or write openatioHj and an axis 
labelled Ec represents the quantity given by equation (23)- Figure 4 is a contour plot of 
the plot of figure 3;, wliere the w and Z axes r^reseoDit the same quantities as the 
respective axes 'm figure 3, The example system iHustiated in the graphs is one in 
which there are 4096 address decoders (i,e. ^=^409^, and data is stored using an 11- 
of-2S6 code (Le. £^11 and J9 ^ 2S6). Other configurations show similar results. 

Referring first to figure 3, it can be seen that tiie number of words which can be 
coneclly retrieved Ec increases as more data is written to memory, (Le. as Z 
inc3rea$es)» but fills off shazply as Z becomes too large (a cliff hke formation 10). 
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That is^ after a number of dai^ mitei, the mCTUucy becomes satoiated, and the number 
of words fhat can be correct recov^ed tUIs ofT sharply. 



Further analysis will now be jnreseated with reference to figure 4, The continuous 
contotxr lines show Hie expected number of correct words varies with the number 
of; stored entries (Z) and 1h.e number of activated address detodeis (wj. The vahie 
of ^i: is 5552, with an error tate of approximately 1 1%. This peak capacity is greater 
than the nnmher of word fines (Ff^096). The central contoiir line 11 is drawn at &c 
s=5000, and wbsequent contcifcrfi are drawn in steps of 500. The south face Of the plot 
is very nearly planar and Ec is very close to Z except at the west side of the plot where 
w i$ small. Performance falls off rapidly if eifter w is to small or Z is to Iaige» as 
shown by die steep slopes at Hie west and north sides of die plot respectively. 

St^erimposed on the continuous contours described above, are two other sets of data. 
A first data set contains three hyperbolae 12 drawn in long-da^hed lines. Each 
hyperbola reiaresents the values of id^ and Z which generate a grven value Syr The 
fbrst of &ese hyperbolae A xtpresrats paiid:s at wlnoh a second hyperbola B 

refuesents A==0L5aiul9thixdhyp^l^ B; can be seen that the Ibie 

denoting optimal occupancy (A=O.S) passes throng the peak; of &e plot 

fiirther curves 13 show how v^ues of wand 2 affect die value of The three 
corves, reading fiom left to li^t r^resrat Pc values of O.I » 0.5 and 0.9 respectively. 
It can be seen from figure 4 that ttis peak of each of ihe curves 13 lies on the line ^ 
O.S. it should be noted that Ibe three curves 1% are close togeCh^ indicatiz^ that 
edacity is relatively insensitive to the probability P^. 

Given the preceding ^scuasion, it can be seen that the optimum v^no for w for 
T na^rfiTiiTm correct dataroDovery^ with other data being recovered with etrar^ occurs is 
at the peak of the plot. That is Mn»l l. The qptimum value jbr error firee recovery is 
read fiom the point at which the central hcyperbola B crosses ihe cssstadl curve of the 
curves 13. (that is the point at which die hr=Q.S line crosses the P^rO.S line). In this 
case iiF=20, which is above twice 'die vahic fbr recovery widi enxM. In order to 




detetmine the value of w for enor firee jeowety^ it is necessary to locate the msxinmni 
of the Pc curve. However, as this curve is very flat detecting the x^aximuin is very 
difScuIt. As it is known Ifast this tnsximum occuis at a point whace A»0.5» it is 
possible to tatce fb& point of intecssdiosa of tiie line P^S and If=OS to detennine the 
Vdhie of ^ fyx maBdmvsxL wxsr free da^ xeoovery. TherBfbre, according to fb& 
developments made by the invCTttion, any memasy ha;vizig Hie pansmeters set out 
above is optimedly operated widi a value ofwbetiveen 11 and 20. 



The priding dzsouasion has proposed a novel way to decetuame an optimal value fi>r 
w and benefits of such a determination have been presented. However, it sbould he 
noted that as each address decode fires indepeodec^tly, if is difScult to ensuce that 
exactly w address are activated by each ioput address, &a£ is fhere will be some 
vaxiation in iv. The analysis presented above is now modified in fbe light of Hds 
observation. 

It has been e^Iained that a threshold Tis applied to an address decoder to determine 
the nmnber of bits that need correspond between an input addn^s and a decoder 
identifier if a decoder is to be activated. The activated address decode are those 
which have at least Tbits of their input couesponding with the input address. 

The probability that a first Ni-oi^M code has T '1 's in common with a second N^-of- 
M code is given by equation (30): 



p^r)^Ui,_^ (30) 



Since fhe address d^x>deRS having T bits in common with. Che icput address are 
activated, the mean valne of -vp^ denotedvi^canbederivednaing equation (31): 



^=~t,<^t-C'^ (31) 
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If tbe m^noiy is opexating wifb enor fi»e input ibe same addiess decodeis will 
always he activated by a given wpxit address. However, there will l7e some stattetical 
vwiation in the number of decoders that are activated by different input addresses. For 
a given ir^ut address, the probability of any one address decoder being acstrvated in 
given by: 



^ (32) 



w 



The W address deoodera give abdnoxniaE distribution for wifb amean. in 
accordance with equation (3 IX aoid a vaziance in accwdance with equation (33): 

<T« ^W-pa-Q-pa) (33) 

Tla^ probability of taking a particular value w ' is: 

= a- ^«)*^"'' -p^"" • ^^^^ 

Equation (13) above gives m e^^ression fcxr lOse probability that all of the smred data 
vataes canbeTBcovcred wilhout enor. Given equation (34)* equation (13) can be 
modified to give: 

ii=nCl-A^f''^'^ (35) 

That i5> equation (13) is znodified to take into account variance in the value of iv. 

Sintikily equatim (23X whidh gives a vahis for the expected number of correctly 
recoverable data items (ignoiing those recovuFed with errors) becomes: 



(36) 
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Given these amendmetots to the values ofEc aad Pc to take nito acconut tbe bincanidl 
distribYitiQa of &e ccsxtour plot of iSgicre 4 is modified to that shown in figure 5. 
The contour lines^ hyperbolae axfd cuiv^ denote the same infomiation as those of 
figore 4. tb& vertical (20 axte of figore 5 represents the same quantity as itae 
respective asds of figure 4. The honzont^ is adjusted so as to reflect the mean value 

In this case, the peak value of i$ lower at jg>=444S, where Z=S440 and \9=1S. This 
peak value no lon^ occurs at A=0.5^ but now occurs at A=0-575. The otot free 
pecfonnance is considerably worse at lower values of w, but the general shape of tibe 
plot is unaffeoted However, it can be seen that givoi the binomial distribullon of 
the value of rxnust be iset as to give w a higher mean vajue than those proposed 
above. The vahie of T required can be detenniaed by xemaagmg equation (31), and 
USmg suitable values &r a, ^, i^andwtodeteririineT, 

The system of flie present invention can be implement^ using artificial neural 
networic technology, and such an implementation is now described. The basic 
stmcture Qf a suitable artificial neural network is shown in figure 6. The neural 
network comprises a layer of input neurons 14 connected to a layer of address decoder 
neurons 15. The data me^noiy comprises D neurons 16. Each data neqron 16 is 
connected to each address decoder nemon IS. Each data neunm 16 has a write data 
input IP and a read data output OP. 

The system is particularly suited to an izoplementation based upon leaky integrate- 
and-fire spiking ncnuons, thzough other neuronal models axe also ^plicahle. An 
integrate and fire neuron has » irqpnts, and takes a weighted sunx of tihese mput& This 
sum is known as the activation level of the neuron. A neiqrra fires only if the 
activation level exceeds a piedetetmined threshold. The implesnentadon described 
below is based upon piking neurons, whicb oul^ a single pulse - that is a neuron 
eifbec &es and generates a pulse or it does not fiie and does mt @»erare a pulse. 
Thus the output of aneuron can be considered to be a digital O/I onfput. The leaky 
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properly of the iiearaxis means fiiat an input U nrn'Titfrined by a nsutcm for a 
predetencined period of time. After fhis time, the actiyxdicui level falls t^acb: towazcls 
zero. Thus, if the inputs JSre in waves, only iiqnits from one givA wave will be taken 
into account in any activation level calculation, aa bapxtis will have '^a!^'' 
back to zero. Tbis *leaky' proper^ is important, as it eiLsarK that anemon does not 
fire mconectly, dne to the eBfccts of previous inputs. The *leaky property ensures that 
ihe neurons wifhin the system settle down into inaodve states between "waves' of 
activity. 

Aa addreffi in Ihc fonn of an i-of-A code is input to the address decoder neurons 15 
firom ttie input neurons 14. This input is a series of / spikes on if of the A input neurons 
to the system, arriving at approximately the same tinxe. These spiking neurons 
contribute to Ihe activation level of some of the address decoder neurons, as is 
describedbelow. 

The address decoder comprises Headdress decoder ncuions 15» Connections between 
fhe input neurons 14 and the neurons associated with fbe addre^ decodeis are fixed. 
Similarly, the threshold at which the address decoders neurons fire is also fixed. For 
^amplcp in Hie sinq)Iified ilhistraiion of figpre 6^ it may be that at least oae of the two 
input nfiqroDS 14 counected to a particular address decoderneuron IS must fire, if that 
address decoder neuron is to fire. That is, each address decoder neuron has alhreshold 
ofl. 

A xnatdbc can be finmed sepresenting the connections between file mpist neurons 14 
and the address decoder neurons IS and dds connectian matrix is complete - that is 
every iiqiut neuron conneots to every addiiess decode neuron. Each address decoder 
neuron has A wei^itSi a weights aro set to T in the connection matrix aud A-^ 
wd^its are set to Part of a connection matrix is shown in figute 7. Here, 
connections ace sbown fibr one adds^ess decoder newcm* Five input lines 17 are each 
ccameeted to a respntive input linje IS of a si^^e address decoder neuron (not 
shown). Eadi of the coxmections between an hiput line 17 and an address decoder 
input line 18 aro allocated ehh^ a T ivdght or a ^0' wdght 




Reading the input lines 17 ftom left to ngal md the addr^s decoder neaHon inputs 
fiom top to l^orttom it can be seen fbat iht connection Isetween the fitst input Hne and 
first address decoder nemon ioput is allocated ^0% wHIe the comieotion between liis 
second izqsqt line and ^ second syBuion input is allocated '1*. 

There axe five input lines 17, and five addiess decoder inputs 18. Connections 
between thiee of thc$& tmes are allocated a weigbi. Thns, each address decodo: 
i^euron is in &ct allocaled a 3-of-S code as an identlfiedr. Far each of the connections 
having a ^1^ wdLght^ an input neuron filing on thfi associated input line canbibiites to 
die activation level of Ifae addr^ decoder nenron. Cocnveifiely, fbr a connection 
having a '0* wdght an input neuron fiixi^ on the associated input line does not 
contribute to the activation levd of diie address i^oder neuron. 

The structure shown in flguie 7 is repeated such that each input line 1 7 is connected to 
W address decoder neuron inpute. la the idew of figaie 7 only a sin^e address 
decoder neurone's inputs are shovn, such that each input line 17 is connected to only 
to a sixigle input 1& 

CSvea that a *r weight means that a firing inpnt neuron will affect an addr^s deoodor 
neuron input 18 (i.e that input line rg)reseiits a *r in the identifier of the address 
decodCT)^ aud that a *0'' is efifeotively a representation of no connection, {Le. a neooa 
firing on this hiput Ihie no effect on the address decoder neuron) rach address 
decoder neuron need only be connected to the i input neurons carrying Ts- This is 
represented in figure 8, where ooly the three mpnt lines 17 having T weights are 
COimected to inputs IS of an address decode neoron IS. 

Thus, the connections between input and address decode neurons csjq be viewed 
either as a. complete coimeetion matrix comprising *0' and 'V values (figure 7), or a 
lonting table containing specific cannections (figure S). These two views are 
equivalent. lEt should be noted that as in the view of figure 7p only connections fitr a 
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sitt^e adtos deooder neuiw 15 are djown in figure 8. 
lie required ftsr each of flie Faddress decoder neuroas. 

m gsueraL Th& ocameotioiia (or * I's) for each address deooder neaiaii are adeofed at 
idodcno. 

men q)ikGS arrive (lepresenting an input address) a nmnber of addisa decoder 
neurons fire. The firing nuficons will be fliose which have recdved *V vahiea on a 
auffident number off weired input lines, so as to raise Ite activation level of the 
addressdecoderneuronabovclhcllireaholdr. llus firing of address decoder neoions 
15 is a w of Wcods as described above. 

The data memory 3 comprises D neurons .16, each of -which has the address 
decoder nenrens 15 as input Each of the i) data memory nemons has a uniqae write 
data input IP. The coimections between the PTaddreas decoder neamns andfteDdata 
memory neurons arc initially set to have zero wei^ts. The coBnections fix>m fte write 
data inpute (not shown) have values which are initiaUy irrelevant Howevs. when a 
write data nearon fires, it must force a respective data memory neuron into a Hd»bian 
learning mode that persists throughout the associated inddent wave of firing address 
dficodar npurons. That is, when data is to be writt^ each of the data memory nenrons 
must be activated sosstobeabletostorediedsta presented by the write data 
xLeunms. 

•n«> ptocedwe flJr writing data to the data memory is now described. Firstly, d of the 
i> ^te data nennms (not shown) fire, and the corresponding d data memory neurons 
16 ara placed in a HeMnan learning mode. The address decoder neurons IS fire as 
described above^ resulting in w of PT code being communiosied to the date memory 
.leoropa W. Hk w neurons receiving spites from the address decoder new>m, and 
^ch have been itoced in HebWan learning mode by the write data 

haveweiglLta setto 




34 



Once flie wave of incident address deoodetr ncaion spikes has passed, the leaky* 
property of all of the neuions causes the aestivation Iwels to reinni to *0'. At this 
poiixt, ttie data, memoiy neurons wase to be in Hebbian learning mode. After a 
suitable delay, liie memory is in a suitable state fer a fiirfher read or write oycle to be 
initialed- Thas it can be seen that the Heaiy' property of the neurons eaisurea that 
activation fiom a previous write cycle do^ not afifect the opeiadon of the current 
write c^L& 

When dsta is to be read fioQi the 6stsL memory; fhe write data neurons are inactive. A 
read opexaticm is initiated by the Sxins of of the IF address decoder neorans as 
desc^bed above. The w fired address decoder neumns cause the activation levels on 
some dala memory neurons to lise, assuming that at least some neurons have bem set 
by aprmous write cycle. 

A gmeiralised ^'tvinner takes (krWIA) style of oidput logic selects df data memoiy 
neurons vnUi thfi highest activalioh level and these issue spikes on their xespectLve 
outpoti^ OP to form a fepres e ni a tiqn of die data output As descaibed above» a suitable 
dday is required before a fbrther read or write cycle is initiated. A nmhber of known 
methods may be used to implement the Jb-WTA alj^tfam. In one such method the 
output nrarons fire in decreasing otdsr of activation level, with the neuron wifli tiie 
hi^est activatton level firing first Thus, aflsar d neurons have fired, it is known that 
^ d neurons having fbe high^ activation levels have oonttibuted to fbQ output. 
Further firing can then be inhibited by a counter neunm which hd$ observed the d 
firing cycles and issues an inhibitory ^gnal to prevent futdiiBX' firings. 

The ^licaiicm of N-of-M codiug to the address decoder logio, as described above 
provides a nmnber of beasts. Particularly, as such cods are interested only in the 
position of *1' bits wtflnn a data value^ they are particularly suited to neural XKtw^ 
applications, where a *1' is presented by a firing neuron- In abinary system it is 
comparatively difHcidt to represent and detect a nenmm not firing makin g a neircal 
implementation of abinary system difficult to realise. Thus, the appUralion of N-of^M 
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coding provides m effective event driven system, where eveoots (anfiuton fires) have 
effeots and soiheveats (anemmdoKxiot&e)donotImvee{&ct& 

Thfiparlicalar selecfion of M andN values fbr aa M-of-N co6a will, to same extent be 
dependeiit upon the E^plica&m. Fw eacdmple, ia bidiifannatics applicadoDS^ where 
DNA data is to be iqpcesented a l-o£4 code is ofim chosen. This choiee is made 
because DNA coniprises a number of bases, each base being selected fiom a set of 
four possible bases, commonly referred to as A» T and C. Thus, each base is 
r^resented by a '1' on one of flie four inputs provided by the code. Ihat is, fl>r 
exaittple: 

G = 0001 A =0010 

T = 0100 C=«1000 

That is a l-of-4 code allows four different values (a two bit binary juzzober) to be 
represented- A DNA sequence is then represented as a saiens of l-af-4 codes* one 
code for each base of Ihe DNA sequisnce* Using this sequence, a four base DNA 
sequence will be represented by a 4-of-l6 code. It should be noted fliat this is a 
restricted -l-of-ie code with a reslriction such that one of the T bits must occur 
withm the first four bits of the code, the second in fiie next &ur bits and so on. 



A general binary code can be transfoimed into an N-of-M cods, &r example using 
dual-rail coding. Hera, eadi bit of &e Wnary code is allocated to two ^'rails". A pulse 
is tiansjDitted on the first rail of a bit if that bit is set to '1\ wWle a pulse is 
liansmitted on the second rail of that bit if fixe bit is set to *0\ Relatiag this dual rail 
impleraentation to the neural network illustrated in figure 6, each iiput neuron 14 
accepts input from two dual rail input nemtms. That is there are ten dual rail input 
neurons in Ihe system. When input is received, this vsoll be in the fbmn of a S-of-IO 
code; wlHi a r^tnctiou such that one '1' nnist occur on one of Ihe fiKSt pair of 
nmnns, one on the sccotkI and so* Input neurons receiving inputs fiom 1he '1' xail 
neutoos then jSre as descdbed above. 
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It should be noted that the additicn of dual rail technology in this way providea a 
system in which the dual rail inputs are operated using a restricted N-of-M code a$ 
described, while tiie input neurons operate in hinaxy, wfti no certainty as to how many 
neuions will 5re, This provides an sSeotive coitveision betwesn an N-of-M system 
and a binary aystem. 

It will be ^parent to those skilled in the art &at the memory ccmfiguratiQn of the 
present invenlaQn may be inapl^ented in terms of traditional address decoders, and 
word lines of data store or as a neural system as described above. It wiU also be 
appazeoot that the ayatem wn be impleixienied either by fabricating bespoke hardware 
componeotap or by cQofigming reconfigiirable components such as Field 
Programmable Gate Arrays (FPGAs). Altemativdy^ the system may be realised fay 
\VEifing a suitable ocHx^ixter program to run on a confVCTtional computer system. Such 
a system will Xkse the hardware of the convetriional confutes- in such a way as to 
simulate a memory embodying the invention* The coni^utear program can be written in 
any of the large number of widdy u$ed coc^mter programmmg languages, alfhou;^ 
an object (»drated implementation is particularly preferred. 
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CLAIMS 



1 A memory oonfigiirad<m use in a oomputer B^^tem, the manary 
comprising a plurality of address decodflrs eacb of widdi is allooflted m identifier 
having a predetenmaed manber of bits, each Mt having first and second selectable 
states, and a data memory having a plmaUty of word lines of predetennioed kngth, 
egch of the said address decoders being activatable to select one of plurality of 
word lines, and the address decoders comptising means to receive an input address 
having a prBdeteminBd number of bits and means to compare the idflriifier of an 
address decoder with the iiqjnt address wherein the memory fiirflier conqjrises means 
to activate an address decoder if at least a predeterairaed minimum nmnber of bits set 
to .the first selectable state in the input address conespond to bits set to the first 
selectable state in &e dscoder idsatifier. 

2. A memojy configurBtion according to claim I. wherein the means to compare 
me identifier of an address decoder with the ii^Jiit address consideis positional 
oonespondenoe between bits set to the first selectable sate in the iiq)iit address and 
bits set to Iha first sdtectable state in^ decoder idcntififira. 

3. A memoiy oonfignradon aocoiding to daim 1 or 2, wherein each address 
decoder identifiiBr has an equal mmibar of bits set to the fiist sdectahle state. 

4. AmeinoryoowBgurationaocorfmgtoanypireoedii«claim,whe^ 

to leceivB an input address is configured to rBoeiro addresses conifflning a 
pfedeteimined nomher of bits set to fiiB first selectable state. 

3. A memoxy configuration, according to claim 3 and 4, wheieiii the 
piedetemiined mmiber of bits set to Ihe first selectaible state in an n^nit address is 
equal to fihe mnnber of bits set to Ae first salectabJe state m each of Ihe address 
decoder idesitifieis. 
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6. A memory canfiguration according to any preceding claim, wlietein the data 
memo^ oco^prises 9. plxcrality of sin^e bit memories, sack fhat each bit of each, word 
line is stored in a'singlebit memoiry. 

7. A mfimqiy cQi:ifi.giiT9iiQn aocoiding to my preceding olaim, wherein the data 
nidmory comprises a data inpiit contaiaing an equal number of bite to ea<di of the 
phnalily of woxd lines. 

8. A memoiy oonfiguraiion aocoixSing to claim 7, finther oompti^g data 
writing means to copy data from the data input line to word lines activated by the 
addiess dacodecs. 

9. A memory configuxation according to claim 7 or wherein the data ixq>ut line 
i& configiired to receive input data ccmtazning a predeteonined number of bits set to 
die first sdectabla state. 

10. A memoiy coniigoration according to any preceding claun, iiirfiier oompising 
means to sum values stOErsd at each bit of word Imes activated by an addr^ decoder 
to generate an acdvatian level value for each bit 

11. A memory con&guiation according to claim 9 and 10. further comprising 
means to generate an output word containing the piedetennired number of bits set to 
the first selectable state. 

12. A memory configuration according to claim 1 1, wheanan the bits get to first 
selectable 3tate in the output are the predetermined numb^ of bits having the highest 
activatLOn level. 
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13. Ainemoiy configoTHtionaccordfagto any^ 

memoiy is inqikmaitBd itsistg a plmsfity of artificial aetaons caanected together to 
fbnn ansaral netnroxic 

14. A memory txmSwaxa&m aooarding to dlaJm 13, -wiifirHii ths plirafity of 
addtess decoders are repiesentBd by a plmaHiy of address decoder neuiofts, and the 
daTf Tw«wn«Ty is T K jvr e seAxttA by a plurality of data neutOM. 

15. A neural netwodc niemixy configinHtioii fbrnse in a con^utBr system, the 
memory comprising a plunility of address decoder ncmons each, of wiacb. is 
cMinected to a iwedetenmned mimibar of input netttons, and a datamemoiy having a 
pfttrality data neurons, eacih of die said address decoder neurons hemg aodvatable to 
select some of die plurality of daia neurons, and die addteas decoder oeimsm 
comprising means to receive a signal r^reseoting a firing of an input nemxm to which 
it is connected, v?iierein an address decoder neuron comprises means to activate data 
nettPOttS if firing signals are received from at le^t a predeteannined miniTnum nnmber 

. 0fii«iUtnE«uxms to T«diichl3bfi address decoder neuron 

16. A method for operating a memory for use in a computer systena, the memory 
compiisiqg a phwality of address decoders each of which is allocated an identifier 
havtag a predetermined Bomber of bits, each bit having JBrst and second selectable 
states, and a data meanory having a plurality of word Knes of pradeteraraied length, 
each of diD said address decoders being activatable to sdect one of the plnrality of 
wad Bnes, wherehi an itqpnt address Ijaving a predetearained number of bits is input 
to the addrras decode, the idaitifiar of an address decoder is compared wifh the inpnt 
address and addi^ decoders are activated if at least a predetermined minimum 
number of bits set to Ihe first selectable state in the itput address correqiond to bits 
set to the first selectable state in^ decoder identifier, 

17. A znefihod accard&^ to daim Id, wherein iapat data is pres^ted at a data 
TBpat of 4)0 data memory and the data is written to word lines activated by the 
aodvaCed uMtess decoders. 
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18. A mdliod accoi^ing to daizn 16 or 17, wherem the predetennined mixdmmn 
nt;rmber of brCs is set suoh that fewer than 100 addi^ decoders are activated by any 
valid input address. 

19. A method accordiiig to claiia li, wherein the predBtarxnined mitiimum 
number of bite is set sucli that &wer fbm 50 address decodeis are activated by any 
vaJid xcpic address. 

20* Ameihod acooxding to claim 19» tvhes^lhe pDededsar^^ number 
of bits is set such that &wer than 20 and more than 11 address decoders are a«tixrated 
by any valid mput address. 

21. A metfaod fiir opt$ix)i$iag the operation of a conqniter romiory wbtcli 
comprises a plurality of address docodeas each of 'which is allocated an idntLfier 
b&ving a prBdstmnined nmnher of bits> each bit having Srst and second selectable 
states, and a data memory having apluraHty of imrd lines of predetennined length, 
ea^ of the said sddress decodexs being activatable to select one of the phorality of 
word liaeSj ttio memikiy further comimsing means to receive an input address, and 
means to activate one or more of (be address decoders if a oomjjaxison between a 
decodes: idesttifier and the iiiput address exceeds a pred^emiined comparison 
tfarefiihold; 

the method comprising detemdning an operationally beneSoial nuniber of 
address decoders to be activated in te^ponse to a valid isprt addressj^ and configuring 
the comparison tfareBhold such that a valid input address will activate a number of 
address decodtsrs substantially equal to &e operalionally beneficial number of address 
decoders to be activated. 

72. A method according to claim 21^, wherein the comparison compares the 
nuznber of bits set to the first selectable state in the input address with itu^ number of 
bit$ set to the first selectable state in each of the address decoder identifiexs. 
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23. A method accoidlng to olaiizL 21 or 22, whrnoiii tbe opetaticaially beneficial 
monber is detemiixLed so as to allow inaximiiin eixor &QQ data reoovay fiom tbe data 
memoEry. 

24. A meOiDd aocoiding to claim 23> wherein the opeiatiozially fameficial ntnafaer 
is dotermfncd using a fimction of the fincm: 

IV is Hie QpeialiQiaaUty'b»eficial nimiber of addiess decodeis to be actlvate«^ 
is the piobability of an arbitrary hit in the data xoanioxy being set to the Srst 
selectable stat^ 

D is the number of bits in each word line, and each wmd line has dhits set to 
the first selectable stale; md 

Po is file probability of a bit bdng corxectly recovened &sm fhe data store; 

25. A nieihod according to claim 24» wherein fiie fimction is: 

26- A method according to claim 21 or 22, wherein fhe operationally beneficial 
number of address decoders to be activated is determined so as to allow mfecrmirm 
error &ee data reoovery,^ while allowing some data to be recovered with errors. 

27. A method accoordii^ to claim 26, whereui the operationally benefidial number 
j$ detennined using a fbnotioa of the form: 

where; 

w IB ttxe cpetatianally beneficial nnmber of address decoders to be activated; 
h is the probability of an arbitrary bit in fhe data memory being set to the first 
sclectaOble state; 
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J3 is the number (rflMts in caiA woid Ime, 
the first Bdectable state. 

28. Am^<rtlaocordnigtodaim27,wherdnfljsfimc1icio 

TV 

29. A method according to daim 21 or 22. wherein the operationaUy benefioial 
mwiber w is set so as to have a valus no less than that given by the equation: 

and no inoie than thilt givea by the equation: 

30. A cairicrmediBm caitying oomputer readable code means to csase a computBT 
to execute piocedme in accardance with the method of any erne of claims 16 to 29. 

31. A COmputBrprognan fcr oarrying out the melhod of anyone of claims 16 to 29. 

32. A memaiy oonfigotatioott substantially as hesjesnbefbce d«caifaed, with 
le&ienoo to figues 2 to 8 of tbe acconqnnymg ^ 

33 . A msSiod fcr operating a mflinory substantiaUy as haidnbefiMB descsribed, 
withxeEferaoee to figures 2 to 8 of fta accompanying diswings. 

34. A nwa«)dfiw optimising file ofperation of a oomputer me^^ 
becembefins described with wfisreoco to figure 3 to 5 of flffl aoconqjanyfaig drttwinga. 
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ABSTRACT 



A memory configuration for use in a ooininiter system, the memoiy congnising a 
plmality of address dscodsrs each of which is allocated an ideadfier having a 
pxed^emdiied nmnber of bits, eadi bit having first and second selectable stetes, an 
data memoty having aptoalily of word Unes of predetermined length* each of Ihe 
said address decodershdng aclivatable to select one of the pluraUty of word lines, 
and flie address decoders oon?>rising means to leodvo an input address having a 
predstemdned number of bits und means to compaie die idcndfier of an address 
decodar witii Ihe input address Ti*jarwi 

activate an address dficoder if a* least a predetetnrined mininnnanrariber of bits se 
flic first adcotaWe state in th© ii^nit address coriespond 
Bslfictable state in the decodec ideadfier. 
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